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Abstract 



In a sample of 19 million produced B mesons we have ob- 
served decays B r]K*, and improved our previous measurements 
of B ^ rj'K. The branching fractions we measure for these decay 
modes are B{B+ rjK*+) = (27.3t|^ ± 5.0) x 10-^ B{B^ 7]K*^) = 
{IS.Stli ± 1-7) X 10-^ B{B+ r]'K+) = (80t^° ± 8) x 10-^ 
and B{B'^ rj'K^) = (88l}g ± 9) x 10"^. We have searched with compa- 
rable sensitivity for related nonstrange decays, and report upper limits for 
these rates. All quoted results are preliminary. 
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The dominant decay modes of B mesons involve the b ^ c quark transition with couphng 
to a boson. For many of these modes the decay amphtude may be described by a tree 
diagram in which the hght quark (spectator) is bound in both the initial B meson and 
final charmed hadron via soft gluon exchange. With recent improvements in experimental 
sensitivity, less favored modes are becoming accessible. These include: h ^ u tree diagram 
transitions that are suppressed by the small Cabibbo-Kobayashi-Maskawa [jl| (CKM) matrix 
element Vuh-, such as B ^ Triu p|; effective flavor changing neutral current (FCNC) decays 
b ^ s described by loop diagrams, such as the "electromagnetic penguin" B — > K*'y [||; 
and decays to charmless hadrons such as B ^ Kir The hadronic decays may be 

classified according to contributions to the amplitude from the several tree and penguin 
diagrams shown in Fig. |l] 0J^. Some of these charmless hadronic decays offer prospects 
for the observation oi CP violation, while others facilitate the quantitative understanding 
of the amplitudes that are essential to the interpretation of future CP measurements. For 
example, the decays B rjK and B tj'K, with B Kn, have been examined in this 
context Ulig. 
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FIG. 1. Feynman diagrams describing the representative decays r]^'^K^*^^: (a, b) internal 

penguins; (c) external tree; (d) flavor-singlet penguin. 



In this paper we present results of several new experimental searches for B meson decays 
to two-body final states containing 7] and rj' mesons. These / = mesons are mixtures of 
flavor-SU(3) octet and singlet states, the latter being of particular interest because of its 
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allowed formation through a pure (two or more) gluon intermediate state (Fig. ^ (d)). 

The data were accumulated at the Cornell Electron-positron Storage Ring (CESR). The 
integrated luminosity was 9.13 fb~^ for the reaction e~^e~ — > T{4S) — >■ BB (center-of-mass 
energy Ecm = 10.58 GeV). This luminosity corresponds to the production of 9.6 x 10^ charged 
and an approximately equal number of neutral B mesons. In addition we recorded 4.35 fb~^ 
of data with Ecm below the threshold for BB production to measure continuum processes. 
These constitute the complete data sample from the CLEO II and CLEO II. V experiments, 
and the measurements we report here succeed our earlier ones |lTT| , p!2| from subsets of these 
data. 

The CLEO II detector [0 emphasizes precision charged particle tracking, with specific 
ionization (dE/dx) measurement, and high resolution electromagnetic calorimetry based on 
CsI(Tl). Scintillators between the tracking chambers and calorimeter provide time-of-flight 
information which we use in conjuction with dE/dx for particle identification (PID). The 
CLEO II. V configuration differs in two respects: the addition of a silicon vertex detector; 
and replacement of the 50:50 argon:ethane gas in the main drift chamber with a 60:40 he- 
lium:propane mixture. From the raw data we reconstruct charged pions and kaons, photons 
(from 7r°, t], and t]' decays), and vr+vr^ pairs that intersect at a vertex displaced from the 
collision point ("vees", from —>■ tt+tt"). Candidate B decay tracks must meet speci- 
fications on the number of drift chamber measurements, goodness of fit, and consistency 
with an origin at the primary or particular secondary vertex. Candidate photons must be 
isolated calorimeter clusters with a photon-like spatial distribution and energy deposition 
exceeding 30 MeV. We exclude photon pairs from extremely asymmetric t] decays, requiring 
|cos^*| < 0.97, where 6* is the meson center of mass decay angle relative to its flight direc- 
tion. This cut rejects soft photon backgrounds and is tightened to 0.90 for K*/p channels to 
veto B K*'y background. We reject charged tracks and photon pairs having momentum 
less than 100 MeV/c. The photon from candidate t]' fr/ decays is generally required to 
have an energy greater than 200 MeV, though this requirement is relaxed to 100 MeV for 
channels with a Ks, which have low background. 

We fit photon pairs and vees kinematically to the appropriate combined mass hypothesis 
to obtain the meson momentum vectors. Resolutions on the reconstructed masses prior to 
the constraint are about 5-10 MeV/c^ (momentum dependent) for 7r° —>■ 77, 12 MeV/c^ 
for Tj — > 77, and 3 MeV/c^ for tt+tt^. Information about expected signal distributions 

with the detector response comes from a detailed GEANT-based simulation of the CLEO 
detector that reproduces the resolutions and efficiencies of data in a variety of benchmark 
processes. In particular, we have established the momentum and dE/dx resolutions in studies 
of K^TT^ decays in a momentum range near 2.6 GeV/c. 

Since the B mesons are formed nearly at rest, while the B daughters we observe are 
relatively light, the latter have momenta close to half of the beam energy (2.6 GeV/c). For 
this reason the final states are well separated from those involving heavier daughters, i.e., 
the dominant b —>■ c decays. The principal signatures for the selected decay modes are 
consistency of the resonance decay invariant masses with the known masses and widths of 
those resonances, and consistency of the total final state with the B meson mass and energy. 
Because the beam energy Eh is better known than the reconstructed B meson energy Eb, we 



substitute the former in the B mass calculation: M = ^E"^ — p'^, with the reconstructed 
B momentum. We define also the variable AE = Eb — E^. The measurement resolution on 
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M is 2.5 - 3 MeV/c^, and on Ai? it is 25-50 MeV, depending on the apportionment of the 
energy among charged tracks and photons for each mode. 

For vector-pseudoscalar decays of the B and the p7 decay of the rj' we gain further 
discrimination from the hehcity variable Ti (cosine of the vector meson's rest frame two- 
body decay angle with respect to its flight direction), which reflects the spin alignment in the 
decay. The decay rate is proportional to IH? when the vector meson decays into two spinless 
daughters, and to 1 — for r/' p7. For modes in which one daughter is a single charged 
track we achieve statistical discrimination between kaons and pions by dE/dx. With Sk and 

deflned as the deviations from nominal energy loss for the indicated particle hypotheses 
measured in standard deviations, the separation Sk — S^, is about 1.7 (2.0) at 2.6 GeV/c for 
the CLEO II (II.V) samples. 

The main backgrounds arise from continuum quark production e+e" qq. We discrim- 
inate against these jet-like events with several measures of the energy flow pattern. One is 
the angle 6bb between the thrust axis (axis of maximum energy projection magnitude) of 
the candidate B and that of the rest of the event. For a fake B candidate selected from 
particles belonging to a qq event, those particles tend to align with the rest of the event, 
whereas the true B decays have a thrust axis that is largely uncorrelated with the tracks 
and showers from the decay of the partner B. We reject events with \cos 6bb\ > 0.9. In 
addition we use a multivariate discriminant JF incorporating the energy deposition in nine 
cones concentric with the event thrust axis, and the angles of the thrust axis and pb with 
respect to the e~^e~ beam direction We have checked the backgrounds from the favored 
B decay modes by simulation, flnding their contributions to the modes in this study to be 
generally quite small. Where appropriate we include this component in the flts described 
below. 

We set the selection criteria for mass, energy, and event shape variables so as to include 
sidebands about the expected signal peaks. To extract event yields we perform unbinned 
extended maximum likelihood flts to these data of a superposition of expected signal and 
background distributions: 

(Ns+Nb) N 

C{Ns, Nb) = n [NsVsiP; X,) + NBVBif, x. 



1=1 



Here Vs and Vb are the probability distribution functions (PDFs) for signal and continuum 
background, respectively. They are functions of observables Xj for event i, and of parameters 
P and 7 (discussed below). The form of C reflects the underlying Poisson statistics obeyed 
by Ns and Nb, the (non- negative) numbers of signal and continuum background events, 
respectively. At the maximum of C, Ns + Nb is equal to the total number of input events. 
Observables for each event include M, AE, J-', and (where applicable) resonance masses and 
Ti. For these flts ranges from ~ 100 to a few thousand. 

For B^ decays |l^ that have a primary daughter charged hadron (generically h~^) that 



can be either vr^ or we flt both modes simultaneously, with C expanded so that the 
signal and background yields of both vr"*" and are flt variables. In this case the PDFs 
depend also on the dE/dx observables S"^ and Sk- The modes with a secondary vector decay 
{K* or p) h^iji^ or vr*^) also require special treatment. For these modes the momentum 
spectrum of is bimodal because of the forward-backward peaked 7i distribution. We 
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TABLE I. Measurement results. The first column lists the final states, with secondary decay 
modes as subscripts. The vector modes with secondary decays to h'^TT^ or /i+vr" are further distin- 
guished with a subscript (— ) or (+) depending on the value of H (see text). The remaining columns 
give event yield from the fit, reconstruction efficiency e, total efficiency with secondary branching 
fractions Bg, significance, and the resulting B decay branching fraction B with upper limit. 



Final state 


Fit events 


6(%) 


eH,(%) 


Signif. {a) 


H(10-6) 


90% UL(10-6) 


n' K+ 


39.6l^° 

'-'-'--10 
q 2+3-6 
2 9 

29.6i:^ 


27 

29 
24 
28 


4.7 

8.7 
1.4 
2.9 


13.4 

10.1 
7.7 
8.9 


88ll« 

79+13 
'^-12 

67i? 
10511 








28 
30 


4.7 
9.0 


0.0 
0.8 


c: 1+8.3 
'^•-'--5.1 


13 
21 




n+6.0 

'J-'J-o.o 
'J-'J-o.o 

^•^-0.0 


45 

29 
38 
25 


17.5 

6.6 
5.1 
1.9 


1.4 
0.0 
0.0 
0.0 


q cr+3.5 
0+3-^ 


10 
9.4 
12 
23 


??37r7r+ 


r. 7+5.7 
"J- ' -4.6 

^•^-0.0 

^•^-0.0 


46 
30 
35 
20 


18.2 
6.8 

13.7 
4.6 


1.3 
0.0 
0.0 
0.0 


q 9+3.3 
"^•^-2.6 

o+°-s 

0+^-^ 


9.2 
7.0 
3.9 
11 


^77-^K+7rO(+) 
^77-^^0^+ 


"•'-'-0.0 
Q n+2.7 


12 
9 
8 
6 
24 
17 


1.6 
1.2 
0.7 
0.5 
2.2 

o.y 


4.0 
0.0 
1.4 
1.9 
2.1 

O 1 

z.l 


"3^-23 

0.0+12 

171?? 

80in 

1(5+14 
0/1+30 


125 
53 
96 
269 

44 

no 
9o 


^77-^*°(-) 

rj^^n {+) 


0-^-3.0 
n ri+2.1 

4.7+0 
2.7li:? 


16 
15 
11 
10 


4.3 
4.0 
1.7 
1.6 


4.2 
0.0 
3.8 
1.0 


16.71?;^ 

n A+5.4 

O-OIo.o 

28111 

1811 


36 
17 
68 
69 


^77 /'^(-) 

r?37rP'^(-) 
r?37rP"^(+) 


o.olJ:^ 

'-'•*-0.4 

Q q+3.2 
0.0_2 1 

l-4l?i 


12 

9 
8 
6 


4.8 
3.6 
1.9 
1.4 


0.0 
0.1 
2.1 
0.6 


o-olO 

1+12 

181}^ 
10l?« 


14 
31 
58 
118 




2.oii:i 

9 q+4.3 
^■'*-2.3 


26 
18 


10.3 
4.2 


1.1 
0.7 


9 n+3.3 
^•'-'-2.0 

61^^ 


10 

28 
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select independent K* and p samples to fit according to the sign of Ti. Events with 7i < in 
our sign convention have low momentum and are unambiguously separated by kinematics 
combined with PID information from dE/dx measurements. For the events with Ti > Q the 
separation is less sharp, so we allow the yields for both hypotheses to float in the global 
likelihood fit and include the PID information. For K*^ / the choice of K or ir is made on 
the basis of dE/dx and time-of- flight information. 

The PDFs Vs and Vb are constructed as products of functions of the observables Xj. The 
dependences of Vs on masses and energies are Gaussian, double Gaussian, or Breit-Wigner 
functions, whose means, widths, etc. appear as the parameters (3 in Eq. |l|. The background 
PDF Vb contains signal-like peaking components in its resonance mass projections, to ac- 
count for real resonances in the background, added to smooth components for combinatoric 
continuum. The smooth components are low-order polynomials, except that for M we use 
an empirical shape |jl5| that accounts for the phase space limit at M = Eb- The dependences 
of both Vs and Vb on JF, Sk, and S-,, are bifurcated Gaussian functions. We obtain the 
parameters j3 oiVs from separate fits to simulated signal, and 7 of Vb from fits to data in 
a sideband region of the AE - M plane. Where the Monte Carlo estimate of background 
from T(4S') production is non- negligible, we add a term with a free fit variable to account 
for this as well. 

Results for all of the B decay chains appear in Table |. The row label subscripts denote 
secondary decays, including rj' rj-K^-K^ with 77 ^ 77 {rjirir), rj' —>■ p'-f {p'~f), and 1] —>■ tt+tt^tt'^ 
(Stt). For each mode the table gives the event yield from the likelihood fit, reconstruction 
efficiency, total efficiency, significance, branching fraction, and (where appropriate) 90% 
confidence level upper limit. Significance is defined as the number of standard deviations 
corresponding to the probability for a fiuctuation from zero to our observed yield. 

Where we have measured a given B decay mode in more than one secondary decay chan- 
nel, we combine the samples by adding the = ~"2 InC functions of branching fraction and 
extracting a value with errors or limit from the combined distribution. The limit is the value 
of B below which 90% of the integral of C lies. We summarize in Table |I| the results for these 
measurements, along with some earlier ones for related decays, and corresponding theoretical 
estimates P,p!7|-[T9[] . We include systematic errors from uncertainties in the PDFs, i.e., in f3 
and 7, obtained from a Monte Carlo convolution of the likelihood function with Gaussian 
resolution functions for these parameters, including their most important correlations. This 
procedure changes the upper limit by less than 10% in most cases. We also include system- 
atic errors for reconstruction efficiencies and selection requirements, and quote upper limits 
computed with efficiencies one standard deviation below nominal. 

We have analyzed each of the decays also without use of the likelihood fit, employing 
more restrictive cuts in each of the variables to isolate the signals. The results are consistent 
with those quoted in the tables, but with larger errors (less restrictive limits) in most cases. 

The positive signals we find in both charge states of B rjK* are first observations: 
B{B+ 7]K*+) = (27.3;^^ ±5.0) x lO'^ and B{B^ t]K*^) = {IS.StH ± 1-7) x lO'^. 
(The first error quoted is statistical, the second systematic.) The significance is about 5 
standard deviations for both, as can be seen in the likelihood functions from the fits shown 
in Fig. 1^. For the B — ^ rjK* decays we show also in Fig. |^ the projections of event distribu- 
tions onto the M axis. The signals appear as peaks at the B meson mass of 5.28 GeV/c^ in 
these plots, as well as in AE and the K* mass. 
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TABLE II. Combined branching fractions. The fourth column gives our prehminary result, 
as central value followed by statistical and systematic error if significantly above zero, otherwise 
as a 90% confidence level upper limit. We quote estimates from various theoretical sources for 
comparison. 



Decay mode 


Sfit(10-6) 


Signif. (fj) 


^(10-6) 


Source 


- 


1 T - 1 


SO+IP + 8 


16.8 


80+3° ± 8 


This 


- 




88+}? + 9 


11.7 


88+}? ± 9 




- 




1 (-1+5.3 _|_ Q 1 


2 






B^ - 


/ fi 






< 11 




, -. 1 

B+ - 






1.2 


< 87 


IpMI 


B^- 






1.0 


< 20 




B+ - 


r]'p+ 






< 47 


Q 










< 23 





B+ - 


r]K+ 


9 9+2.6 
^ 2 2 


1.0 


< 7.1 


This Exp. 


B^- 


r]K° 




0.0 


< 9.5 


This Exp. 


B+ - 


r/vr"*" 


1 9+2.6 


0.6 


< 6.0 


This Exp. 


B^- 


r]7r^ 




0.0 


< 3.1 


This Exp. 


B+ - 


r]K*+ 


27.3t|^ ± 5.0 


4.8 


27.3l|^ ± 5.0 


This Exp. 






13.8l|^ ± 1.7 


5.1 


13.8111 ± 1.7 


This Exp. 


B+ - 


r]p+ 


4.3^3:^ ±0.7 


1.3 


< 16 


This Exp. 


B^- 




2.6i|^ ±0.3 


1.3 


< 11 


This Exp. 



Theory ^(10" 



7-65 |13,|19| 



9-59 [17,19| 



1 - 23 |13,|19| 
0.1 - 14 |]jl| 
0.1 - 3.7 [0jl|] 
0.1 - 8.0 [01111 
3-24 0,1191 
0.1 - 11 0Jil 
0.2 - 5.0 PJl 
0.1 - 3.0 
1.9 - 7.4 
0.2 - 4.3 
0.2 - 8.2 
0.1 - 8.9 
4 - 17 |1^| 
0.1 - 6.5 [llTHH 
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FIG. 3. Projections onto the variable M. Overlaid on each plot as smooth curves are the best 
fit functions (solid) and background components (dashed), calculated with the variables not shown 
restricted to the neighborhood of expected signal. The histograms show (a) i?^ r]K*^; (b) 
rjK*^. The unshaded histograms represent the Ti > K* samples and the shaded is the rest. 



With the full CLEO data sample we also improve our previous measurements |Tl| , p^ 
of i? — > Tj'K. The new branching fractions are B{B^ rj'K^) = (SOlg^ ± 8) x 10"^ and 
13{B° 7]'K°) = {88t\l ± 9) X 10-6. The likelihood functions from the fits for B r]'h+ 
and B^ rj'K^ are shown in Fig. ^. For these modes we show also in Fig. ^ the projections 
of event distributions onto the M axis. 

The observed branching fractions for B —>■ rj'K and B 1]K*, in combination with the 
upper limits for the other modes in Table |I| and with recent measurements of i? ^ Kn, 
TTTf 1^, and B ^ Tip |^D[, provide important constraints on the theoretical picture for these 
charmless hadronic decays. 

The effective Hamiltonian calculations ||^ commonly used to account for the charmless 
hadronic B decays contain uncertainties in form factors l21|j2^ , light quark masses ||2^ , and 



the QCD scale. They generally employ spectator and factorization approximations, with 
unknown color octet terms arising from the expansion parameterized by an effective number 
of colors. 

Two approaches, with many variations, to understanding these decay rates have been 
proposed. A large ratio of {B 7]'K,rjK*) to {B riK,ri'K*), consistent with our mea- 
surements, was predicted in terms of interference of the two penguin diagrams in Fig. 
|I|(a) and (b), constructive for r]'K and riK* and destructive for riK and rj'K*. The argument 
is a bit more complicated for t]K* than rj'K, and there are probably cancellations that off- 
set the enhancement. With independently estimated form factors the calculations tend to 
predict smaller rates than we observe. 

The second class of ideas, which address particularly the large rj'K rate, take the t]' to be 
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FIG. 5. Projections onto the variable M. Overlaid on each plot as smooth curves are the best 
fit functions (solid) and background components (dashed), calculated with the variables not shown 
restricted to the neighborhood of expected signal. The histograms show (a) 'q'h'^ with 

T]' ryvrvr (ry Svr, dark shaded), r/' ryvrvr (r/ 77, light shaded), and 77' — > fry (open); (b) 

— > r]'K^ with r]' — > t/tttt (shaded) and ry' — > p7 (open). 
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a special, poorly understood object with an anomaly contribution (Fig. |1| (d)) in constructive 



interference with the penguins |1T0|J25| , |26[| . This is related to speculations regarding the gluon 
| 25| or cc content [^,0 of the r( . These are not equivalent; the anomaly term is a specific 
contribution which has been calculated (and shown to be insufficient to explain the data). 

In combination, our riK* and riK observations suggest that perhaps the penguin inter- 
ference mechanism is more important than current estimates give, with unexpectedly large 
form factors or some other mechanism favoring the penguin terms for both modes. The 
models that focus entirely on enhancement mechanisms for the r/' appear to be insufficient 
to explain all of the data. 
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